Unpacking the dark variance of differential attainment on examinations in overseas graduates.

CONTEXT: Differential performance in postgraduate examinations between home medical graduates and those who qualified outside their country of practice is well recognised. This difference is especially marked in the practical component of the UK Membership of the Royal College of General Practitioners (MRCGP) examination. The potential causes of such disparities are not well understood. METHODS: Data were available for 1874 international medical graduates who applied for general practice (GP) specialty training in the UK in 2008-2012. The primary outcome was performance in the Clinical Skills Assessment (CSA) OSCE component of the MRCGP. The main predictors were performance on a situational judgement test (SJT) and clinical problem-solving test (CPST), a test of applied clinical knowledge, used in the selection for GP training. Data relating to demographic characteristics and English language fluency were also available. To better understand the relationship between the predictors, the selection measures and the outcome, a series of univariable and multivariable models were developed and tested, concluding with a structural equation model to explore causality. RESULTS: The CSA rating was more strongly predicted by SJT scores (standardised beta, 0.26) than by performance on the CPST (standardised beta, 0.17). There was a relationship between English language fluency and CSA score that was mainly mediated via SJT performance. CONCLUSIONS: These findings demonstrate that performance on an SJT predicts performance in a high-fidelity clinical simulation (the CSA) in international medical graduates. Although the constructs tested by SJTs are debated, and are likely to vary across settings, culturally appropriate knowledge of interpersonal competence is likely to be evaluated. Improving the confidence of doctors in this area through targeted educational interventions, rather than focusing on increased clinical knowledge, is likely to be more effective at reducing disparities observed in postgraduate examination performance. Thus, there are important implications for the design of specialty selection and licensing assessments globally.

An exploration of applicant perceptions of asynchronous video MMIs in medical selection.

This article was migrated. The article was marked as recommended. Over the last two decades, technological advancements internationally have meant that the Internet has become an important medium for recruitment and selection. Consequently, there is an increased need for research that examines the effectiveness of newer technology-mediated selection methods. This exploratory research study qualitatively explored applicant perceptions of fairness of asynchronous video interviews used in medical selection. Ten undergraduate medical students participated in a pilot asynchronous multiple-mini interview and were invited to share their experiences and perceptions in a follow-up interview. The data was transcribed verbatim and analysed using template analysis, with Gilliland's (1993) organisational justice theory guiding the original template. Many of the original themes from Gilliland's model were uncovered during analysis. Additionally, some significant themes were identified that did not form part of the original template and were therefore added to the final coding template - these were specifically relating to technology, including acceptability in a medical context; technical issues and adverse impact. Overall, results suggested that participants perceived asynchronous video interviews to be a fair method of selection. However, participants thought asynchronous interviews should only be used as part of an extensive selection process and furthermore, should not replace face-to-face interviews. Findings are discussed in line with existing research of fairness perceptions and justice theory in selection ( Gilliland, 1993) and implications for research and practice are presented.

Development of a questionnaire to evaluate patients' awareness of cardiovascular disease risk in England's National Health Service Health Check preventive cardiovascular programme.

BACKGROUND: The National Health Service (NHS) Health Check is a cardiovascular disease (CVD) risk assessment and management programme in England aiming to increase CVD risk awareness among people at increased risk of CVD. There is no tool to assess the effectiveness of the programme in communicating CVD risk to patients. AIMS: The aim of this paper was to develop a questionnaire examining patients' CVD risk awareness for use in health service research evaluations of the NHS Health Check programme. METHODS: We developed an 85-item questionnaire to determine patients' views of their risk of CVD. The questionnaire was based on a review of the relevant literature. After review by an expert panel and focus group discussion, 22 items were dropped and 2 new items were added. The resulting 65-item questionnaire with satisfactory content validity (content validity indices≥0.80) and face validity was tested on 110 NHS Health Check attendees in primary care in a cross-sectional study between 21 May 2014 and 28 July 2014. RESULTS: Following analyses of data, we reduced the questionnaire from 65 to 26 items. The 26-item questionnaire constitutes four scales: Knowledge of CVD Risk and Prevention, Perceived Risk of Heart Attack/Stroke, Perceived Benefits and Intention to Change Behaviour and Healthy Eating Intentions. Perceived Risk (Cronbach's α=0.85) and Perceived Benefits and Intention to Change Behaviour (Cronbach's α=0.82) have satisfactory reliability (Cronbach's α≥0.70). Healthy Eating Intentions (Cronbach's α=0.56) is below minimum threshold for reliability but acceptable for a three-item scale. CONCLUSIONS: The resulting questionnaire, with satisfactory reliability and validity, may be used in assessing patients' awareness of CVD risk among NHS Health Check attendees.

A preliminary investigation to explore the cognitive resources of physicians experiencing difficulty in training.

BACKGROUND: Treating patients is complex, and research shows that there are differences in cognitive resources between physicians who experience difficulties, and those who do not. It is possible that differences in some cognitive resources could explain the difficulties faced by some physicians. In this study, we explore differences in cognitive resources between different groups of physicians (that is, between native (UK) physicians and International Medical Graduates (IMG); those who continue with training versus those who were subsequently removed from the training programme); and also between physicians experiencing difficulties compared with the general population. METHODS: A secondary evaluation was conducted on an anonymised dataset provided by the East Midlands Professional Support Unit (PSU). One hundred and twenty one postgraduate trainee physicians took part in an Educational Psychology assessment through PSU. Referrals to the PSU were mainly on the basis of problems with exam progression and difficulties in communication skills, organisation and confidence. Cognitive resources were assessed using the Wechsler Adult Intelligence Scale (WAIS-IV). Physicians were categorised into three PSU outcomes: 'Continued in training', 'Removed from training' and 'Active' (currently accessing the PSU). RESULTS: Using a one-sample Z test, we compared the referred physician sample to a UK general population sample on the WAIS-IV and found the referred sample significantly higher in Verbal Comprehension (VCI; z = 8.78) and significantly lower in Working Memory (WMI; z = -4.59). In addition, the native sample were significantly higher in Verbal Comprehension than the UK general population sample (VCI; native physicians: z = 9.95, p < .001, d = 1.25), whilst there was a lesser effect for the difference between the IMG sample and the UK general population (z = 2.13, p = .03, d = 0.29). Findings also showed a significant difference in VCI scores between those physicians who were 'Removed from training' and those who 'Continued in training'. CONCLUSIONS: Our results suggest it is important to understand the cognitive resources of physicians to provide a more focussed explanation of those who experience difficulties in training. This will help to implement more targeted interventions to help physicians develop compensatory strategies.

Selecting for creativity and innovation potential: implications for practice in healthcare education.

The ability to innovate is an important requirement in many organisations. Despite this pressing need, few selection systems in healthcare focus on identifying the potential for creativity and innovation and so this area has been vastly under-researched. As a first step towards understanding how we might select for creativity and innovation, this paper explores the use of a trait-based measure of creativity and innovation potential, and evaluates its efficacy for use in selection for healthcare education. This study uses a sample of 188 postgraduate physicians applying for education and training in UK General Practice. Participants completed two questionnaires (a trait-based measure of creativity and innovation, and a measure of the Big Five personality dimensions) and were also rated by assessors on creative problem solving measured during a selection centre. In exploring the construct validity of the trait-based measure of creativity and innovation, our research clarifies the associations between personality, and creativity and innovation. In particular, our study highlights the importance of motivation in the creativity and innovation process. Results also suggest that Openness to Experience is positively related to creativity and innovation whereas some aspects of Conscientiousness are negatively associated with creativity and innovation. Results broadly support the utility of using a trait-based measure of creativity and innovation in healthcare selection processes, although practically this may be best delivered as part of an interview process, rather than as a screening tool. Findings are discussed in relation to broader implications for placing more priority on creativity and innovation as selection criteria within healthcare education and training in future.

Measuring resilience in palliative care workers using the situational judgement test methodology.

OBJECTIVES: Relatively little research has been directed toward the assessment of resilience in the health care context. Given the stressors associated with the provision of health care, the present study describes the development and evaluation of a situational judgement test (SJT) designed to assess resilience in palliative care health care workers. METHODS: An SJT was developed to measure behaviours associated with resilience in a palliative care context. Next, SJT reliability and validity analyses were assessed in a sample of acute ward, hospice and community palliative care workers (n = 284). RESULTS: Findings showed the SJT to have a negative association with self-reported sickness absence and a positive association with well-being and employee attitudes (in terms of turnover intention, organisational commitment and job satisfaction). A series of two-wave longitudinal (hierarchical) regressions showed the SJT was predictive of well-being and employee attitudes at two time-points (4 weeks apart) over and above self-report measures of resilience, education and experience, and the Big Five personality dimensions of Emotional Stability and Openness. Reliability analyses showed the SJT to have acceptable test-retest scores (ρ = 0.71) and high internal consistency (α = 0.91). CONCLUSIONS: The study findings suggest that the SJT is a valid assessment of resilience in at-risk workers and can be used in either summative or formative assessment under the right set of conditions. More research is needed to test the hypothesis that the SJT might be used as a formative tool to develop workplace resilience.

Recruiting for values in healthcare: a preliminary review of the evidence.

Displaying compassion, benevolence and respect, and preserving the dignity of patients are important for any healthcare professional to ensure the provision of high quality care and patient outcomes. This paper presents a structured search and thematic review of the research evidence relating to values-based recruitment within healthcare. Several different databases, journals and government reports were searched to retrieve studies relating to values-based recruitment published between 1998 and 2013, both in healthcare settings and other occupational contexts. There is limited published research related to values-based recruitment directly, so the available theoretical context of values is explored alongside an analysis of the impact of value congruence. The implications for the design of selection methods to measure values is explored beyond the scope of the initial literature search. Research suggests some selection methods may be appropriate for values-based recruitment, such as situational judgment tests (SJTs), structured interviews and multiple-mini interviews (MMIs). Personality tests were also identified as having the potential to compliment other methods (e.g. structured interviews), as part of a values-based recruitment agenda. Methods including personal statements, references and unstructured/'traditional' interviews were identified as inappropriate for values-based recruitment. Practical implications are discussed in the context of values-based recruitment in the healthcare context. Theoretical implications of our findings imply that prosocial implicit trait policies, which could be measured by selection tools such as SJTs and MMIs, may be linked to individuals' values via the behaviours individuals consider to be effective in given situations. Further research is required to state this conclusively however, and methods for values-based recruitment represent an exciting and relatively unchartered territory for further research.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100.

Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals' reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

Resilience through the lens of interactionism: a systematic review.

This systematic review presents findings from a conceptual and methodological review of resilience measures using an interactionist theoretical framework. The review is also intended to update findings from previous systematic reviews. Two databases (EBSCOHost and Scopus) were searched to retrieve empirical studies published up until 2013, with no lower time limit. All articles had to meet specific inclusion criteria, which resulted in 17 resilience measures selected for full review. Measures were conceptually evaluated against an interactionist framework and methodologically reviewed using Skinner's (1981) validity evidence framework. We conclude that inconsistencies associated with the definition and operationalization of resilience warrant further conceptual development to explain resilience as a dynamic and interactive phenomenon. In particular, measures of resilience may benefit from a greater focus on within-person variance typically associated with behavioral consistency across situations. The use of alternative measurement modalities to self-report scales, such as situational judgment tests, is proposed as a way of advancing knowledge in this area.

Development of competency models for assessors and simulators in high-stakes selection processes.

BACKGROUND: Selection for entry into UK medical specialty training is a high-stakes, high-volume process. For selection into General Practice, a large number of assessors and simulators are involved in the delivery of the selection centre, which represents the final stage of selection. AIM: In order to standardize and quality-assure assessor and simulator involvement in the process, we developed two competency models outlining the knowledge, skills and attributes associated with each role using a previously validated job analysis methodology. RESULTS: The final qualitative analysis resulted in two competency models, each encompassing eight competency domains. In general, results from a validation questionnaire demonstrated positive feedback from various regional recruitment leads in the UK (n = 14). CONCLUSION: Both models are currently being used in practice for quality assurance and training purposes. We conclude that the competency models can be used in three ways: (1) recruiting assessors/simulators; (2) in measuring performance of assessors/simulators and highlighting areas for potential development; and (3) they can be used for training assessors/simulators.

Evaluations of situational judgement tests to assess non-academic attributes in selection.

OBJECTIVES: This paper presents a systematic review of the emerging international research evidence for the use of situational judgement tests (SJTs) for testing important non-academic attributes (such as empathy, integrity and resilience) in selection processes. METHODS: Several databases (e.g. MEDLINE, PsycINFO, Web of Science) were searched to retrieve empirical studies relating to SJTs published between 1990 and 2010. Personal contact with experts in the field was made to identify any unpublished research or work in progress to obtain the most current material. Finally, reference lists were checked to access other relevant journal articles and further research. All research studies were required to meet specific inclusion criteria selected by two independent reviewers. RESULTS: Over 1000 citations were identified during the initial literature search; following the review of abstracts, full-text copies of 76 articles were retrieved and evaluated. A total of 39 articles that adequately met the inclusion criteria were included in the final review. The research evidence shows that, compared with personality and IQ tests, SJTs have good levels of reliability, predictive validity and incremental validity for testing a range of professional attributes, such as empathy and integrity. CONCLUSIONS: SJTs can be designed to test a broad range of non-academic constructs depending on the selection context. As a relatively low-fidelity assessment, SJTs are a cost-efficient methodology compared with high-fidelity assessments of non-academic attributes, such as those used in objective structured clinical examinations. In general, SJTs are found to demonstrate less adverse impact than IQ tests and are positively received by candidates. Further research is required to explore theoretical developments and the underlying construct validity of SJTs.

Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection.

OBJECTIVES: This study aimed to evaluate the validity and utility of and candidate reactions towards cognitive ability tests, and current selection methods, including a clinical problem-solving test (CPST) and a situational judgement test (SJT), for postgraduate selection. METHODS; This was an exploratory, longitudinal study to evaluate the validities of two cognitive ability tests (measuring general intelligence) compared with current selection tests, including a CPST and an SJT, in predicting performance at a subsequent selection centre (SC). Candidate reactions were evaluated immediately after test administration to examine face validity. Data were collected from candidates applying for entry into training in UK general practice (GP) during the 2009 recruitment process. Participants were junior doctors (n = 260). The mean age of participants was 30.9 years and 53.1% were female. Outcome measures were participants' scores on three job simulation exercises at the SC. RESULTS: Findings indicate that all tests measure overlapping constructs. Both the CPST and SJT independently predicted more variance than the cognitive ability test measuring non-verbal mental ability. The other cognitive ability test (measuring verbal, numerical and diagrammatic reasoning) had a predictive value similar to that of the CPST and added significant incremental validity in predicting performance on job simulations in an SC. The best single predictor of performance at the SC was the SJT. Candidate reactions were more positive towards the CPST and SJT than the cognitive ability tests. CONCLUSIONS: In terms of operational validity and candidate acceptance, the combination of the current CPST and SJT proved to be the most effective administration of tests in predicting selection outcomes. In terms of construct validity, the SJT measures procedural knowledge in addition to aspects of declarative knowledge and fluid abilities and is the best single predictor of performance in the SC. Further research should consider the validity of the tests in this study in predicting subsequent performance in training.

Evaluating candidate reactions to selection practices using organisational justice theory.

OBJECTIVES: This study aimed to examine candidate reactions to selection practices in postgraduate medical training using organisational justice theory. METHODS: We carried out three independent cross-sectional studies using samples from three consecutive annual recruitment rounds. Data were gathered from candidates applying for entry into UK general practice (GP) training during 2007, 2008 and 2009. Participants completed an evaluation questionnaire immediately after the short-listing stage and after the selection centre (interview) stage. Participants were doctors applying for GP training in the UK. Main outcome measures were participants' evaluations of the selection methods and perceptions of the overall fairness of each selection stage (short-listing and selection centre). RESULTS: A total of 23,855 evaluation questionnaires were completed (6893 in 2007, 10,497 in 2008 and 6465 in 2009). Absolute levels of perceptions of fairness of all the selection methods at both the short-listing and selection centre stages were consistently high over the 3years. Similarly, all selection methods were considered to be job-related by candidates. However, in general, candidates considered the selection centre stage to be significantly fairer than the short-listing stage. Of all the selection methods, the simulated patient consultation completed at the selection centre stage was rated as the most job-relevant. CONCLUSIONS: This is the first study to use a model of organisational justice theory to evaluate candidate reactions during selection into postgraduate specialty training. The high-fidelity selection methods are consistently viewed as more job-relevant and fairer by candidates. This has important implications for the design of recruitment systems for all specialties and, potentially, for medical school admissions. Using this approach, recruiters can systematically compare perceptions of the fairness and job relevance of various selection methods.

New machine-marked tests for selection into core medical training: evidence from two validation studies.

This study examined whether two machine-marked tests (MMTs; a clinical problem-solving test and situational judgement test), previously validated for selection into U.K. general practice (GP) training, could provide a valid methodology for shortlisting into core medical training (CMT). A longitudinal design was used to examine the MMTs' psychometric properties in CMT samples, and correlations between MMT scores and CMT interview outcomes. Independent samples from two years were used: in 2008, a retrospective analysis was conducted (n=1711), while in 2009, CMT applicants completed the MMTs for evaluation purposes (n=2265). Both MMTs showed good reliability in CMT samples, similar to GP samples. Both MMTs were good predictors of CMT interview performance (r = 0.56, p < 0.001 in 2008; r = 0.61, p < 0.001 in 2009) and offered incremental validity over the current shortlisting process. The GP MMTs offer an appropriate measurement methodology for selection into CMT, representing a significant innovation for selection methodology.